A performance analysis framework for SOCP algorithms in noisy compressed sensing

Solving under-determined systems of linear equations with sparse solutions attracted enormous amount of attention in recent years, above all, due to work of \cite{CRT,CanRomTao06,DonohoPol}. In \cite{CRT,CanRomTao06,DonohoPol} it was rigorously shown for the first time that in a statistical and large dimensional context a linear sparsity can be recovered from an under-determined system via a simple polynomial $\ell_1$-optimization algorithm. \cite{CanRomTao06} went even further and established that in \emph{noisy} systems for any linear level of under-determinedness there is again a linear sparsity that can be \emph{approximately} recovered through an SOCP (second order cone programming) noisy equivalent to $\ell_1$. Moreover, the approximate solution is (in an $\ell_2$-norm sense) guaranteed to be no further from the sparse unknown vector than a constant times the noise. In this paper we will also consider solving \emph{noisy} linear systems and present an alternative statistical framework that can be used for their analysis. To demonstrate how the framework works we will show how one can use it to precisely characterize the approximation error of a wide class of SOCP algorithms. We will also show that our theoretical predictions are in a solid agrement with the results one can get through numerical simulations.Comment: arXiv admin note: substantial text overlap with arXiv:1303.729

Discrete perceptrons

Perceptrons have been known for a long time as a promising tool within the neural networks theory. The analytical treatment for a special class of perceptrons started in seminal work of Gardner \cite{Gar88}. Techniques initially employed to characterize perceptrons relied on a statistical mechanics approach. Many of such predictions obtained in \cite{Gar88} (and in a follow-up \cite{GarDer88}) were later on established rigorously as mathematical facts (see, e.g. \cite{SchTir02,SchTir03,TalBook,StojnicGardGen13,StojnicGardSphNeg13,StojnicGardSphErr13}). These typically related to spherical perceptrons. A lot of work has been done related to various other types of perceptrons. Among the most challenging ones are what we will refer to as the discrete perceptrons. An introductory statistical mechanics treatment of such perceptrons was given in \cite{GutSte90}. Relying on results of \cite{Gar88}, \cite{GutSte90} characterized many of the features of several types of discrete perceptrons. We in this paper, consider a similar subclass of discrete perceptrons and provide a mathematically rigorous set of results related to their performance. As it will turn out, many of the statistical mechanics predictions obtained for discrete predictions will in fact appear as mathematically provable bounds. This will in a way emulate a similar type of behavior we observed in \cite{StojnicGardGen13,StojnicGardSphNeg13,StojnicGardSphErr13} when studying spherical perceptrons.Comment: arXiv admin note: substantial text overlap with arXiv:1306.3809, arXiv:1306.3980, arXiv:1306.397

Spherical perceptron as a storage memory with limited errors

It has been known for a long time that the classical spherical perceptrons can be used as storage memories. Seminal work of Gardner, \cite{Gar88}, started an analytical study of perceptrons storage abilities. Many of the Gardner's predictions obtained through statistical mechanics tools have been rigorously justified. Among the most important ones are of course the storage capacities. The first rigorous confirmations were obtained in \cite{SchTir02,SchTir03} for the storage capacity of the so-called positive spherical perceptron. These were later reestablished in \cite{TalBook} and a bit more recently in \cite{StojnicGardGen13}. In this paper we consider a variant of the spherical perceptron that operates as a storage memory but allows for a certain fraction of errors. In Gardner's original work the statistical mechanics predictions in this directions were presented sa well. Here, through a mathematically rigorous analysis, we confirm that the Gardner's predictions in this direction are in fact provable upper bounds on the true values of the storage capacity. Moreover, we then present a mechanism that can be used to lower these bounds. Numerical results that we present indicate that the Garnder's storage capacity predictions may, in a fairly wide range of parameters, be not that far away from the true values

Block-length dependent thresholds in block-sparse compressed sensing

One of the most basic problems in compressed sensing is solving an under-determined system of linear equations. Although this problem seems rather hard certain $\ell_1$-optimization algorithm appears to be very successful in solving it. The recent work of \cite{CRT,DonohoPol} rigorously proved (in a large dimensional and statistical context) that if the number of equations (measurements in the compressed sensing terminology) in the system is proportional to the length of the unknown vector then there is a sparsity (number of non-zero elements of the unknown vector) also proportional to the length of the unknown vector such that $\ell_1$-optimization algorithm succeeds in solving the system. In more recent papers \cite{StojnicICASSP09block,StojnicJSTSP09} we considered the setup of the so-called \textbf{block}-sparse unknown vectors. In a large dimensional and statistical context, we determined sharp lower bounds on the values of allowable sparsity for any given number (proportional to the length of the unknown vector) of equations such that an $\ell_2/\ell_1$-optimization algorithm succeeds in solving the system. The results established in \cite{StojnicICASSP09block,StojnicJSTSP09} assumed a fairly large block-length of the block-sparse vectors. In this paper we consider the block-length to be a parameter of the system. Consequently, we then establish sharp lower bounds on the values of the allowable block-sparsity as functions of the block-length

Upper-bounding ℓ1\ell_1-optimization weak thresholds

In our recent work \cite{StojnicCSetam09} we considered solving under-determined systems of linear equations with sparse solutions. In a large dimensional and statistical context we proved that if the number of equations in the system is proportional to the length of the unknown vector then there is a sparsity (number of non-zero elements of the unknown vector) also proportional to the length of the unknown vector such that a polynomial $\ell_1$-optimization technique succeeds in solving the system. We provided lower bounds on the proportionality constants that are in a solid numerical agreement with what one can observe through numerical experiments. Here we create a mechanism that can be used to derive the upper bounds on the proportionality constants. Moreover, the upper bounds obtained through such a mechanism match the lower bounds from \cite{StojnicCSetam09} and ultimately make the latter ones optimal.Comment: arXiv admin note: text overlap with arXiv:0907.366

Negative spherical perceptron

In this paper we consider the classical spherical perceptron problem. This problem and its variants have been studied in a great detail in a broad literature ranging from statistical physics and neural networks to computer science and pure geometry. Among the most well known results are those created using the machinery of statistical physics in \cite{Gar88}. They typically relate to various features ranging from the storage capacity to typical overlap of the optimal configurations and the number of incorrectly stored patterns. In \cite{SchTir02,SchTir03,TalBook} many of the predictions of the statistical mechanics were rigorously shown to be correct. In our own work \cite{StojnicGardGen13} we then presented an alternative way that can be used to study the spherical perceptrons as well. Among other things we reaffirmed many of the results obtained in \cite{SchTir02,SchTir03,TalBook} and thereby confirmed many of the predictions established by the statistical mechanics. Those mostly relate to spherical perceptrons with positive thresholds (which we will typically refer to as the positive spherical perceptrons). In this paper we go a step further and attack the negative counterpart, i.e. the perceptron with negative thresholds. We present a mechanism that can be used to analyze many features of such a model. As a concrete example, we specialize our results for a particular feature, namely the storage capacity. The results we obtain for the storage capacity seem to indicate that the negative case could be more combinatorial in nature and as such a somewhat harder challenge than the positive counterpart.Comment: arXiv admin note: substantial text overlap with arXiv:1306.3809, arXiv:1306.397

Bounds on restricted isometry constants of random matrices

In this paper we look at isometry properties of random matrices. During the last decade these properties gained a lot attention in a field called compressed sensing in first place due to their initial use in \cite{CRT,CT}. Namely, in \cite{CRT,CT} these quantities were used as a critical tool in providing a rigorous analysis of $\ell_1$ optimization's ability to solve an under-determined system of linear equations with sparse solutions. In such a framework a particular type of isometry, called restricted isometry, plays a key role. One then typically introduces a couple of quantities, called upper and lower restricted isometry constants to characterize the isometry properties of random matrices. Those constants are then usually viewed as mathematical objects of interest and their a precise characterization is desirable. The first estimates of these quantities within compressed sensing were given in \cite{CRT,CT}. As the need for precisely estimating them grew further a finer improvements of these initial estimates were obtained in e.g. \cite{BCTsharp09,BT10}. These are typically obtained through a combination of union-bounding strategy and powerful tail estimates of extreme eigenvalues of Wishart (Gaussian) matrices (see, e.g. \cite{Edelman88}). In this paper we attempt to circumvent such an approach and provide an alternative way to obtain similar estimates.Comment: acknowledgement footnote adde

Lifting/lowering Hopfield models ground state energies

In our recent work \cite{StojnicHopBnds10} we looked at a class of random optimization problems that arise in the forms typically known as Hopfield models. We viewed two scenarios which we termed as the positive Hopfield form and the negative Hopfield form. For both of these scenarios we defined the binary optimization problems whose optimal values essentially emulate what would typically be known as the ground state energy of these models. We then presented a simple mechanisms that can be used to create a set of theoretical rigorous bounds for these energies. In this paper we create a way more powerful set of mechanisms that can substantially improve the simple bounds given in \cite{StojnicHopBnds10}. In fact, the mechanisms we create in this paper are the first set of results that show that convexity type of bounds can be substantially improved in this type of combinatorial problems.Comment: arXiv admin note: substantial text overlap with arXiv:1306.376

Asymmetric Little model and its ground state energies

In this paper we look at a class of random optimization problems that arise in the forms typically known in statistical physics as Little models. In \cite{BruParRit92} the Little models were studied by means of the well known tool from the statistical physics called the replica theory. A careful consideration produced a physically sound conclusion that the behavior of almost all important features of the Little models essentially resembles the behavior of the corresponding ones of appropriately scaled Sherrington-Kirkpatrick (SK) model. In this paper we revisit the Little models and consider their ground state energies as one of their main features. We then rigorously show that they indeed can be lower-bounded by the corresponding ones related to the SK model. We also provide a mathematically rigorous way to show that the replica symmetric estimate of the ground state energy is in fact a rigorous upper-bound of the ground state energy. Moreover, we then recall on a set of powerful mechanisms we recently designed for a study of the Hopfield models in \cite{StojnicHopBnds10,StojnicMoreSophHopBnds10} and show how one can utilize them to substantially lower the upper-bound that the replica symmetry theory provides.Comment: arXiv admin note: substantial text overlap with arXiv:1306.3764, arXiv:1306.397

Under-determined linear systems and ℓq\ell_q-optimization thresholds

Recent studies of under-determined linear systems of equations with sparse solutions showed a great practical and theoretical efficiency of a particular technique called $\ell_1$-optimization. Seminal works \cite{CRT,DOnoho06CS} rigorously confirmed it for the first time. Namely, \cite{CRT,DOnoho06CS} showed, in a statistical context, that $\ell_1$ technique can recover sparse solutions of under-determined systems even when the sparsity is linearly proportional to the dimension of the system. A followup \cite{DonohoPol} then precisely characterized such a linearity through a geometric approach and a series of work\cite{StojnicCSetam09,StojnicUpper10,StojnicEquiv10} reaffirmed statements of \cite{DonohoPol} through a purely probabilistic approach. A theoretically interesting alternative to $\ell_1$ is a more general version called $\ell_q$ (with an essentially arbitrary $q$). While $\ell_1$ is typically considered as a first available convex relaxation of sparsity norm $\ell_0$, $\ell_q,0\leq q\leq 1$, albeit non-convex, should technically be a tighter relaxation of $\ell_0$. Even though developing polynomial (or close to be polynomial) algorithms for non-convex problems is still in its initial phases one may wonder what would be the limits of an $\ell_q,0\leq q\leq 1$, relaxation even if at some point one can develop algorithms that could handle its non-convexity. A collection of answers to this and a few realted questions is precisely what we present in this paper